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ABSTRACT 

Although Fred Keller’s personalized system of 
instruction (PSI) has had a surprisingly rapid adoption in colleges 
and universities since its introduction in 1964, sound evaluation 
techniques for the method are still needed. Verbal reports are vague, 
while other methods that have been used are idiosyncratic and depend 
on the way the teacher presents the course. Oral interviews and 
self- pacing studies have other flaws. Much of the research to date 
has been sloppy, and many procedures have been introduced with only 
the flimsiest bit of empirical justification. Since a primary aim of 
PSI is to stimulate interest in the course, some dependent variables 
with measure interest might be studied. These might include the 
number and pattern of withdrawals in the course, a question-asking 
profile, or use of an interest library consisting of articles related 
to lectures which are stocked in student dormitories. Also, a 
self-management project might supplant traditional rat experiments in 
psychology courses. (JK) 
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^iost often complainte about the adoption of an educational innovation 
are that the adoption ia too slow. In a recent article in Science, Lawrence 
Grayson states, 

"Education requires long lead tines to turn innovation 
into widespread practice. In the 1930's, it was esti- 
mated that 57 years would be required to diffuse an 
innovation throughout the school system, even after the 
materials and methods had been thoroughly developed" 

(1972, p. 1219). 

lie goes on to say that by 1946, the lead time had decreased to 25 years but 
even this can hardly be characterized as rapid change. This tine scale does 
not seem to apply to the educational innovation introduced by Fred Keller, 
called a Personalised System of Instruction (PSl). PSl has been growing at an 
incredible rate. 

Since PSl *8 early trials at the University of Brasilia in 1964 and at 
Arizona State University in 1965 to the present is only 7 or 8 years. Hclilchael 
(1971) surveyed all of the four year institutions in the U.S. that had at 
least three department members listed in the i\PA directory and asked them if 
they were using PSl in any of their courses. Of those returning the questionnaire, 
138 schools (31Z) Indicated that they were using PSl in at least one course, 
lie also found another 23 institutions that were using PSl in departments other 
tlian psychology and in which the psychology department was not using it. Taken 
together, about 37Z of the 440 colleges aii universities surveyed had at least 
one department using PSl techniques. This is an average adoption rate of between 
23 and 27 institutions a year idiich, I'm sure you'll agree, is remarkable. 

Other evidence of widespread acceptance is that a Newoletf.er consisting 
of information about PSl courses, workshops, and new adoptions has already been 
put out. There was 400 requests for tlie newsletter even before the first issue 
was released in Juns, of 1971, and the mailing list for the Newsletter has now 



Jumped to 1000 (PSl Wewsletter, January, 1972). Such intarest and ready accep- 
tance Indicates that moat teachers already know that there is a great deal 
wrong with traditional educational practice, and that they are eager to try a 
pronising new method. 

But whatever the reasons for this willingness to adopt PSl by college 
teachers, 1 wish to inject a caution. Perhaps the adoption has been too fast. 

1 am not alone iu this concern Ben Green at HIT is also worried about the speed 
with which PSl has been catching on (Green, 1971a). VIhyl Well, there is a real 
danger that without sound evaluative research, PSl will go the way of so many 
other attempts to renovate education. Widespread acceptance, of a technique 
without aound evidence for that technique is Just the formula for producing another 
"educational fad." Recent newspaper reports have told the story of how the 
Office of Education has become disenchanted with "performance contracting" by 
private companies as a means of helping the educationally disadvantaged get 
"caught up " (Salt Lake Tribune, i^til 30, 1972). PSl or bastardizations of 
it may be tried and dropped because of an incomplete understanding of Che vari- 
ables contributing to its efficacy, iiy purpose, then, is to point out what 1 
believe to be some problems with the current research; to discuss some research 
that we are doing at Utah State; and to make a few suggestions about the direction 
of future research. 

Some Problems 

Since Keller's (1968) "Goodbye Teacher" article, research in PSl has been 
almost completely restricted to showing that it is superior to traditional 
teaching methoda in producing better learning (Sheppard and ^lacUemot, 197 j; 
llcllichael and Corey, 1969) and. better retention (Cooper and Greiner, 1971). These 
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three articles did use objective exaas to make their comparisons and we do need 
more research to show that PSl is a better way, especially because of the long 
history (1924 to 1965) of "no difference" results when different educational 
techniques have been compared (Dubin and Taveggrio, 1968). llost often, though, 
researchers have relied upon grade distributions and a heavy dose of student 
reports to make their case, l/hat's wrong with this? First, verbal reports, 
while they are suggestive and may provide insight for the teacher, are no 
substitute for observing what the student does with respect to the material, as 
all of us operant conditioners are so fond of pointing out to our cognitive 
brethen. Second, grades are notoriously poo : dependent measures of performance 
especially when the comparison is made across classes. And when we consider 
that the criteria for attaining an A, B, or F, differ drastically from the PSl 
class to the lecture class, then it is t^ruly Impossible to know what different 
grade distributions mean. 

We continue to engage in these comparisons, I suppose, because it is rein- 
forcing to us to plot distributions and repeatedly come up with more "A"s and 
"B"s in our classes than "C"s, "D"s, or "F"s. It also looks good to our dean 
who is always scouting around for "good" instructors and "better" methods of 
instruction. All those "A"s must mean that you are doing something right, and 
of course, students do not find anything to criticize in the lopsided distribu- 
tions either. 

Another trend in the current research that creates serious problems of 
evaluation is that each "pioneer" tends to develop his own version of Keller's 
basic plan and to discourse at length on the idiosyncratic details of his course 
to such an extent that even for tl e "initiates" it is difficult to discover 
what was done and how. The rs'xder is laden with detail upon detail and when he 



d 



•4- 



flnlshes the article he flues that the author simply concludes that "my" PSI 
course was superior to a conventionally taught course. Johnston and Penrt/packcr 
(1971), In describing tlielr PSI course, devoted almost one-thlrd (31%) of a long 
article to procedures. One Is left to wonuer v;hy all tiiat detail was Included, 
or whac there was In the procedures that cor.trluuted to the success of the PSI 
course. 

Plnally, when one considers the coinplexlty, detail, and hence work Involved 
In setting up a course similar to that espoused by a zealous author, one 
wonders why the measures of beuavlor, (learning or Interest In the field) remained 
so gross and subjective. If one is to do all that work, surely he suould be 
convinced that the procedures are related to the outcome and gross measures such 
as grades are not convincing. Jo we really need all those tests and variations 
of tests, or a ratio of exactly lU students to 1 proctor, or doomsday contin- 
gencies, or required attendance? Is the rate at which a student works Important? 
Jo we need to plot Individual cumulative records of the student's progress? 

We will never know until we start treating each of these details as variables 
and manipulating them in a system^.rlc fashion. 

A recent experiment (Stalling, 1971) highlights the dllcnna of the would- 
be adopter of the PSI approach and Illustrates the points I have been making. 

The autiior's purpose \/as to desaonstrate the effectiveness of an overhead 
projector In giving and correcting quizzes ilr-eh~l^ge class and he did. The 
experimental group scored hlgiier than the control on an objebtlve exam and rated 
their course higher tlian did tne controls. But th^^se results were achieved In 
spite of substantial deviations from PSI procedures, nach unit was one chapter 
In kendler's text, so the units were much larger than units used In most PSI 
courses. 'I'ne course was Instructor-paced rather than student-paced. The subjects 
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could taUe all three forms of the unit oulzzcs with only the iilglicst score 
counting. Thus, mastery of the uaterlal was not required. Flnallyi the course 
did not use any proctors. So on most of the Important characteristics of PSI 
courses, the procedures differed, yet tue results favored the experimental group 
and the autaor labeled the course a "programmed course". It Is hard to know how 
to categorize this resaerch. I think It would be unfair and mlsrepresentatlve to 
Include It In the body of research supporting PSI as a better method. I do not 
want to single out this piece of research as being good or bad just that It 
siiould not be classified as PSI research. There .::re significant deviations and/or 
addenda from Keller's original five points In almost every piece of published 
research to date. Let's look at some variables that have been modified, added, 
or left uncontrolled. 

Oral Interviews 

Ferster (196d) has described one variant of PSI which utilized oral Inter- 
vlevrs. His report Is primarily descriptive but he does advocate, via implication, 
that the oral intervle\« Is worth adopting so I will examine his report as If he 
iiad evaluated this Innovation. Ferster 's major dependent variables are student 
reports, grade distributions, and the rate of completion of course materijal. He 
found that most of his st’idents earned A's and tliat 36% finished the course three 
weeks early. The students reported that 1) their verbal fluency and ease In 
social situations Improved; 2) they were able to draw upon their persotial experi- 
ence and relate It to the material In the unit; and 3) their study habits Improved 
In other classes. 

Now, V7lth respect to the loajor Independent variable of the study, oral Inter- 
views, what can be said? Very little Is supplied to the reader which would permit 
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hlm to decide for nlmself wliether be should adopt the technique or not. The 
student reports about verbal fluency and ease In social situations have an 
obvious face validity but It would be desirable to have Independe ;t support for 
this finding. Tlie report about students' Improved study skills generalising to 
other classes Is potentially a very Important finding and deserves experimental 
substantiation. From the data supplied, It Is difficult to Infer what was 
responsible for the favorable reports or to determine the degree to which the 
students' verbal behavior corresponded to other performance criteria. 

We have tried oral Interviews at USU and are currently relying on them as 
one major component In our Introductory course. On the face of It, oral lntervlet- 7 S 
permit rapid feedback, an opportunity to correct misunderstandings In a relatively 
nonthreatening way (a student proctor corrects another student), and an opportunlf'' 
to probe when the proctor suspects that the Interviewee does not "really" under- 
stand the material in question, but It Is more difficult to maintain quality 
control with oral interviews than with written quizzes. Also some proctors are 
obviously much more Interested In and willing to help their charges than others. 
With a written quiz, the stimulus presented to each student Is constant. 

Obviously, there are advantages and disadvantages to each method. Wotlce, however, 
that we have adopted an Innovation, oral interviews, without evidence of its 
superiority over the original method. 1 confess to the same unsystematic tinkering 
that 1 am now condemning. As far as 1 know, there lias been no critical study 
comparing the oral to the written quiz (but see Wlnkel, 1971). 

Self-pacing 

Ferster presents individual cumulative records of Interviews taken and the«« 
data show that students work at different rates. The modal student produced a 
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"scalloped" pcrfonoance mucn like wiiat we have come to expect when traditional 
methods of Instruction are used. Viils finding contradicts data reported by 
Lloyd and i^nudsen (l‘>69) who found that once a student started working, he worked 
at a constant and hlgn rate (the break and run performance found under fixed 
ratio schedules). In Lloyd et al's study, good students, those who end up wlt.i 
A's, started to work earlier t.ian mediocre and poor students. Since, "poor" 
students do not complete as much work as good students, they earn a loi;er grade. 

We have tried "doomsday ' contingencies to get the sluggards going and tills 
has generated a "scalloped" perfoiviance In most students. We are no using an 
adjusting schedule \ 7 hlch gives more points the earlier the work Is done. This 
sciiedule has proved much more effective In getting students started and It also 
eliminates the scalloped performance (Cheney and Powers, 1971). Since In most 
PSI courses there arc contingencies piled upon contingencies. It is difficult to 
account for any given pattern of taking exams. It Is obviously premature to 
Characterize performance under PSI contingencies (whatever these are) as one way 
or another. 

Beyond the confusion over the pattern of responding, there Is the finding 
that some students flnlsa the course early. For example, during the fall quarter 
of 1971, we used the adjusting point contingency within a self-paced program and 
20% of the class finished the course wltuln 7 weeks. Having students finish early 
Is touted as In soiue way beneficial to the student and as evidence for the superi- 
ority of PSI over traditional Instruction. There Is little Icnowledge about tiie 
relationship beti/ecn the rate of doing work and the quality of the learning. In 
fact, rate of work may not be related to learning. Johnston and Pennypacker (1971) 
couqiared a group who had to respond at a fixed, high rate of correct responding and 
a low rat«i of incorrect responding when taking a test to a group who had no 






temporal constraints Imposed and founu tuat tlicre was no difference In the quality 
of the learning, iiotii groups responded at better than the level on their 
measure of performance. The autiiors do not report data on the number of students 
finishing the course early In each group so It was not known how the quality of 
learning was related to rate of progress throug.i the course. The question remains 
Jo those who flnlsn early learn more or retain It longer than those who complete 
the course In the ailJ.oted time? Of course, if they do not, what advantage accrues 
to the technique of self-pacing? Perhaps the only advantage Is motivational -- 
It Is reinforcing to flnlsii early. Certainly, our students seem to t/.lnk so. 

To the question, "The best tnlng about this class was ...", many of our students 
ansx/er that ic was the opportunity to finish early. If this Is the primary 
benefit of self-pacing, then we should be apprised of this fact and should not 
overworry about an Individual's rate of progress. In fact, Greenspoon (PSI 
dewsletter, June, 1971/ reports tnat at Temple huell over 50% of the courses 
are taught on an Individual rate basis with no time limitation. The Instructor 
simply notifies the registrar wnen the student has met the objectives of the 
course. If the rate of progress does not matter to any learning variable, uhy 
bother to record tne cumulative rate of Interviews taken. If all yxmr students 
are not finishing your course, you probably arc trying to cover too much 
material but It does not take Individual cumulative records to discover this. 

Jnlt Size 

A variable which 1 believe Important Is the size of the unit. Within PSI 
courses, it has varied from a chapter In a standard introductory text (Cooper and 
Greiner, 1971; Stalling, 1972} to a specially written section of two or three 
pages (Ferster, 1968). This difference In the amount of textual material covered 






per unit ooviously maucs it difficult to compare the rcsuJ.ts of PSI courses at 
different schools, iliio diificuity Is furt.ior roi>ipouiu.cd when one discovers 
tiiat special projects and other tasks are Included iu tiie material teat define: 
a uiiit (Lloyd and Knudsen, 1^169: liertens, 19/0; ilalott and Svlnlckl, 1969). One 
would certainly expect that unit size and difficulty would determine how many 
students finish the course and what percentage finish early but so far very few 
reports have concerned tliemselves ultii the amount of v;ork required per unit. 
Burnstcln's recent article (1972) Is one exception. He continually decreased 
the amount of material over a four-year period as students uncovered difficult 
material. It was Interesting t..at althOw^h t..e units senurully got "easier**, the 
average final exam performance remained unchanged at about 80%. In fact, none 
of his several procedural changes produced Improved learning to a significant 
degree. Wliat did change was that tne students rated the course more favorably. 
Students can be made to do work harder. It seems, but they will resent It. 

Cray Osborne and I are preparing a ter.t to accompany the PSI course at USl> 
and we have decided on about 5 to 7 pages of textual material per unit. Tills 
"bite size", as we call It, was decided arbitrarily with not much empirical 
support except that we noted that the amount of material In most Introductory texts 
was too long and contained too mucii Information co demand the kind of mastery 
that a PSI course is built upon, .i'ow. It w'ould seem tiiat a programmed text would 
be just what Is needed In a PSI course but these have problems too. Students 
frequently complain that these texts are "dull and tedious." These Impressions 
concur with my o\m and argues against them for Inducing excitement and Interest 
In psychology, which is one of our objectives at USU. Again, however, these are 
Subjective judgements and certainly s'uould not be the major criterion by which 
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tlie size of the "bite" Is fixed. Uhere Is the research taat states that 5 to 
7 pages of text or 5 to 7 concepts Is optimal and Is all that should be Included 
In u unit? 

A related question concerns the nature and number of questions asked over 
the unit. Since ve attempt to promote application and Integration of knowledge, 
we ask only a few (G to 10) questions and Include one or two "think" questions 
after each unit. This question format Is similar to Ferster and Perrot's (1968). 

On the other hand, some authors (e.g., hlialey and Halott, 1971) ask numerous, 
flll-ln and true-false questions. In the Whaley and Malott text, for example, 
some units have more than 100 questions, a truly Impossible number If you use 
oral exams. Again, there Is little researca to support either approach. 

i 

/ 

Lectures ^ j 

One of our objectives In the Introductory course Is to stimulate Interest 
In psychology. To accomplish this, wc liave scheduled some lectures by several 
faculty members on topics which we felt to be provocative and Interesting. We 
were hoping, by using lectures as entertainment, to generate questions about, 
and Interest In, the topic discussed and through generalization to the field of 
psychology. This has not proved very successful. For a variety of reasons, 
attendance at these lectures was low. Wlien no contingencies were placed on 
lecture attendance and It was made clear to the student that ".he material presented 
would not uelp him pass the course material, very few students attended. Of a 
class of 180 this quarter, from 5 to 18 students have attended when no other 
contingencies were In effect. There were some who appeared Interested and a few 
asked questions after the lecture was over but there was the usual 3 or 4 who 
slept through the entire lecture. However, when one point was occasionally 
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offered for attendance (a very small payoff In our point system), attendance 
regularly jumped to 40-50. It Is difficult to conclude from our experience 
with lectures that they are "relnforcers" for most students. 

Psychology Is only one of several courses which t-he student is taking so 
lie has plenty of opportunity to attend boring lectures In other courses. Ills 
experience with these required lectures may condition him to respond to all 
lectures as aversive events. One should remember that the program one designs 
for the student In psychology Is only a part of the student's program. Until 
all courses use lectures as vehicles for stimulating Interest and retain Individ- 
uals who are good at oral presentations, lectures will not be relnforcers for 
most students. 

Ilore to the point, if one believes that "Interest lectures". are beneficial 
to students, one should design an experiment to demonstrate this. It Is difficult 
to evaluate the teclinlque In which students have to have so many points or have 
passed X number of units before they can attend a lecture. Green (1971b) found 
that the rate of test taking Increased just prior to the first lecture in his 
course and he noted some spontaneous comments from students Indicating that they 
wanted to complete some material so that they could attend. But, as he admitted, 
there were confounding factors operating and he could not conclude that students 
would work to attend a lecture. One could require that students do a special unit 
that was not required for the course In order to attend a lecture. If a student did 
do the unit, one could rest assured that the reluforcer was the forthcoming lecture 
and not the completion of required material which may be reinforcing by Itself. Al- 
ternately, one could charge X number of points for attendance although the big pro- 
blem here Is that an Individual has to endanger his grade (depending on the relatl\'e 
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cost lectures) to attend. 

Ah B urni ng^ for the moment, that lectures can be arranged which are Interestlr.^ 
anti that i t can be shown tliat some students will work or pay to attend, why should 
we Include this relnforcer In our course? It Is not because students will learn 
from lectures. We are down on the lecture method Just because students do not 
learn from It! So we must hope to Interest students In the topic of discussion 
and eventually psychology. If t' Is Is our Intention, and It seems plausible, we 
better start measuring whatever effects lectures are having. Is our objective 
to have students indicate on some measurement scale that they found the lecture 
Interesting? Although such an Indication by the students may be reinforcing to 
tl'e lecturer, 1 am sure most of us want to accomplish more tiian this. I suspect 
that most of us believe tliat If the student la Interested In the field, he will 
learn more, take more courses In the field and/or adopt the field as a minor or 
major. If these are what we hope to accomplish by Including stimulating lecture; 

In our course, then we must demonstrate these putative effects. This research is 
not easy. It Is difficult to measure the relationship between "lectures and 
Interest" or between "interest" and "use of principles" one, two, or three yc ,rs 
later. But there Is no other way of knowing v;hether we ara engaging In "super- 
stitious" behavior or not. 

To sum up, I think It is fairly obvious to you by now that much of the research 

to date has been sloppy, and that many procedures have been Introduced with only 

the fllmsyest bit of empirical Justification. Keller has attempted to caution the 

users of his system to tiilnk more carefully about the consequences of their changer;, 

but his warning has not been taken seriously. He writes, 

"Generally speaking, no change should be Introduced within a 
programmed system without carefully considering Its possible 
effect upon everyone Involved. It Is not enough to think only 





13 - 



of the student's welfare; the proctor, the assistant, and the 
instructor must also be looked after, if their optiiaiil inter- 
action is to be uiaintained, and if the system itself is to 
survive. What is meat for one must not be poison for another" 

(1971, p. 321). 

At this point, I would like to concentrate on some ideas about the future direc- 
tions of FSI research. 

Self-management projects 

One objective of our course at USU is to try to show students that the 
princ. .c they have been learning really work. That, 1 would suppose, is usually 
why a lab is added to a course. We have tried using animals and special events 
such as a rat Olympics but there are problems with animal projects. First, with 
4U0 students and two students to a rat, you need 200 rats. This requires more 
space and more money than we have. Second, there is a large percentage of students 
that are not very interested in animals or have an active dislike of rats. Third, 
the majority of students, while they find the animal performances "cute", do not 
believe that the principles you used to train them have any relevance to human 
be'navior. Finally, animals are messy and I have been in trouble with both janitors 
and deans because my rats did not respect their carpets. 

We are currently using a self-management project which we feel has been 
successful in teaching students how to apply behavioral principles and in devel- 
oping the students' interest in the subject matter. The details of this project 
are described elset/here (Edwards and Powers, 1971) but the general plan is to have 
the student keep a 2-week baseline on some self-behavior and then attempt to 
modify this behavior by applying a consequence. The student keeps daily graphs 
and summaries and turns in a weekly report which is graded and returned to him. At 
the end of the quarter, he turns in a complete paper sunmariziug his attempt to 
change his behavior. 
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From our standpoint, the project has been successful: students appear keenly 
Interested in their o:/n behavior and most seem to make a genuine attempt to 
follow the guidelines we establish for chp.nglug their behavior* The comments 
that students report In their papers Is also reinforcing to us. Students say such 
things as: "1 didn't believe 1 could ever stop biting my nails but 1 did" or "1 

am going to continue using these techniques to completely eliminate my bad habit". 
The general tenor of these comnents Is that the student did not believe that the 
techniques would work at the outset but now he or she is convinced that they do 
and is glad to have been taught "something useful". On a scale from 1 to 10 with 
10 being the most Interesting feature of the class, the students gave the self- 
management projects a modal rating of 9. 

We have developed an Information and attitude survey which we give at the 
beginning and end of our course. One item asks the student to estimate the 
probability that he or she will become a psychology major (on a scale from 1 to 
10, very unlikely to very likely) if they are not already psychology majors. 

The first quarter that we Introduced the PSl course was difficult for us and 
confusing to the students and we found that there was shift away from becoming a 
psychology major. This quarter, the fifth PSl quarter, we found the opposite 
trend. About 44% of the students shifted In the direction of becoming a psychology 
major while about 27% shifted away (see table 1). Figure 1 shows the percentage 
of students who found the PSl course "less", "equally" and "more" Interesting 
and informative than their other classes for the fall quarter of 1970 and again 
one year later. There is an encouraging shift towards a more favorable assessment 
by the students. This indicates, perhaps, that we are doing something right and 
we believe that the self-management projects had much to do with enlivening the 
course and changing the percentage of students who shifted toward becoming majors. 
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Uowever, we made too many other changes to be able to conclude this. We are 
currently trying to design a study which will permJ t us to assess the effects of 
the self -management projects but It Is still In preliminary stages so I will not 
go Into It. 

An Interest library 

A special library could be maintained by a department In which books were 
catalogued In terms of Interest and difficulty level. Ilovlng the library or parts 
of It Into the dorm Is currently being tried In another Institution and Douglas 
(1971) reports Increased student usage of such a library. A psychology library 
could be arranged so that records would be kept on which students checked out 
what books. The student "librarian" could evaluate what the student learned 
quickly and easily by administering a short test and the student could earn some 
bonus points for extra reading. In this way, an Interest profile could be 
obtained for all students using the library. The amount of reading as well 
the rate of use might provide useful Indexes of Interest created by various PSI 
techniques. Thus, If one wanted to knov; to what extent a "motivational" lecture 
on Imprinting piqued students' Interest, he could measure the Increment In the 
number of articles checked out In the area of ethology In the first few days after 
the lecture. The library x>rould have to have many duplications, say 50 to 100 
copies of a given article, but Its holdings would not need to be extensive. There 
Is also no reason why the unit of material checked out need be a book. In fact, 
having large numbers of books In long stacks too closely reseouiles a conventional 
library and might not engender much usage. The unit checked out could be as small 
as a one-page magazine article. 

It may be countered that what Is wanted Is a good reader. Dut we liave little 
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control over how much material is read In a reader unless we require It and then, 
of course, we have no way of knowing now much of the material. If any, would have 
been read because the student's interest lias been aroused. 



Developing a Question-asking repertoire 

One of the often-heard comments by educators is tliat tiie ability to ask good 
questions Is the mark of a good student, thinker, or scientist. Vet, we pay 
little attention to generating question-asking In our students. We expect our 
graduate students to learn to ask insightful and provocative questions by simply 
sitting In lectures and listening to their professors. Gordon Flamner summarizes 
the problem succintly, 

"There Is time for only a limited number of questions, and 
these are only by the more agg.'esslvc students. What about 
the students who are afraid they will make themselves look 
foolish by asking a "dumb" question? So they rationalize 
that after class they will look up In the text what they 
didn't understand in the lecture. But these people are In 
trouble generally and are behind. They don't have time to 
look up very many questions and so they just add to their 
already vast supply of cumulative Ignorance" (1971, p. 9). 

We have begun to collect some data on the asking of questions In a large 
classroom. For three quarters in 1970-71, my assistant and I wore a wrist 
counter and counted every oral question or comment made to us by the students, 
mst of the time, questions were asked during the class period but occasionally 
questions were asked at other times. We Included such mundane questions as, "Will 
there be a final?" or "Who Is my proctor?" so the count was generous. There 
was many more opportunities to ask questions than In an ordinary course because 
only one day a week was given to a lecture or demonstration and there were two 
"instructors" to ask quehtlons^ of . 

Figure 2 shows the number of question asked In the winter quarter, 1971 and 
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It can be seen tnat about 33 questions or comments were made a day. There Is 
some tendency for the questions to taper off as the quarter progresses and on a 
few days (days before written review exams) the number of questions Increased. 
The numbers beside some of the data points Indicate the number attending at the 
beginning of the class. This day was a llonday and was the day on which the 
lecture or demonstration was scheduled. It appears that there Is very little 
correlation between the number In attendance and the number of questions asked. 
Figure 3 shows the number of questions asked one of our better proctors and 
indicates that she received between 10 and 13 questions a day. Point "a” is a 
day when both the Instructors were absent from the class and point "b" was the 
day Just before a big review exam. 

We also solicited written questions from our students and varied the number 
of points given per question. Students could submit any number of questions 
with the only requirement being that the question had to deal with some aspect of 
psychology. Figure 4 shows that we received about 20 questions per day when the 
questions were worth one point and about 5 times that amount when 2 points per 
question were given. Hov/, what can be made of this information on question- 
asking? 

Wlien one considers that there were over 200 students registered for the 
class, one is struck by the fact that not much behavior was generated by our 
procedures. If we were to assume that each student asked one question a day, 
either oral or written, we would expect to have about 200 questions per day, 
and this goal was rarely obtained. Furthermore, one question a day from each 
student is not very much behavior especially if one Intends to shape the asking 
of good questions. 

One Inference from this data is that there is more response cost to the 
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asklng of questions, even la private (i.e., written questions), than we had 
expected. The implication is obvious: the traditional lecture class with a single 

Instructor doing most of the talking leaves, no opportunity to shape the asking of 
questions. Uov;, if it is true that learning to ask numerous, good, questions 
leads to more and better learning, then we should start to design programs which 
shape these skills. The proctor systeia that characterizes PSI appears especially 
well— suited for such a program. With a ratio of 10 students to 1 proctor, the 
task of reinforcing individual students for asking questions would not be too 
demanding. 

Currently, we are engaged in an experiment which V7lll permit us to determine 
the effects of writing questions on learning. The basic design is an ABA in which 
during tne experimental conditions, students write down questions as they study. 

A proctor records the students' study time, reads his questions, and gives the 
student points based on the quality of the questions. Under control conditions, 
the student does not write questions. Hopefully, this study should tell us the 
value of asking questions as one studies and whether the '-'quality" of a question 
can be specified to the extent that other students can shape "good" questions. 

Summary and Conclusion 

In summary, PSI is being tried at a very fast pace and much of the PSI research 
has been poor. As Born (1971) has pointed out, we do not really know what tradi- 
tional educational techniques do, so research on PSI is made more difficult 
because one does .not know what "baseline" to compare its effects to. When one 
adds to these difficulties the confusing procedural menus, intentional and uninten- 
tional, that are found in the recent literature, one feels like throwing up one's 
hands . 
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I have suggested several dependent measures which might prove useful In 
evaluating various PSI procedures. Thus» an Interest library might provide a 
behavioral laeasure of Interest generated by a course or some part of a course. A 
related measure of interest mlgiit be the number of students who Indicate an 
Increased likelihood that they will major or minor In psychology. Of course » 
this expression of verbal Interest should be validated by actual count but this 
Is not a difficult problem. 'Je might attempt to define what constitutes a good 
student and begin to shape up these requisite skills, although this procedure does 
not constitute a part of PSI per se. Question-asking seems to be one such skill 
and there Is no reason why a four-year profile might yield much more relevant 
Information than grades or, even, which courses the student took, especially 
to the potential employer who is seeking a good scientist or problem-solver. The 
bonus to the researcher is that he might use the number and quality of questions 
students ask as a dependent measure for an Independent variables of his Interest. 

Other variables, such as the number and pattern of withdrawals In the course, 
could help us diagnose trouble spots and may uncover hidden assumptions that we 
are making, nave Born, at the University of Utah, has found that three times 
as many students wlthdrex<7 from his PSI course than from a traditionally taught 
course and that most of the withdrawals came after the student had passed the 
last unit. He also found, contrary to what you might expect, tliat It was the 
"poor" students who were withdrawing (Born, 1971; Born and Herbert, 1971). These 
findings are somewhat surprizing and Indicate, I think, a failure In our 
programming. To exchange C's, D's, and F*s for withdrawals Is not a satisfactory 
solution for dealing with poor students especially. If those students wlthdrai; 
with a bad taste In their mouth. 
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Finally, 1 believe chat many of us would like to Chink Chat what we 
Caught in our introductory course had some relatively lasting effect. After all, 
the course is about behavior, their behavior, and nothing would seem more 
convincing to the student that to be able to use the Information he learned in 
class to bring about a change in his life. So if we would like to claim that 
some technique such as oral interviews or a lab project helped the student learn 
to study, we should measure study skills for students who receive such treatment 
and those who do not. This seems too obvious to have to say but, unfortunately, 
these simple comparisons have rarely been made. Fred Keller has given us an 
exciting and effective way Co help students learn. The PSI approach deserves 
better treatment than it has been getting at the hands of its friends. 





Table 1 



Wumber of students indicating change in probability of becoming 
a Psychology major after taking PSI course, Fall Quarter, 1971. 





Change away from becoming 


No change 


Change toward becoming 




Psychology Major 




Psychology Major 










Total 


N 


43* 


53 


77* 


173 


X 


25 


31 


44 


100 



*P ^.01; Sign test 
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